Skip to content

Conversation

@jason810496
Copy link
Member

related: #59747

Why

It would be really nice to make breeze k8s dev as ONE STEP command to setup Airflow on Kubernetes development environment!

Additionally, while testing out the new breeze k8s dev --dags-path files/dags --deploy command locally, I encounter some errors:

  1. ImagePullBackOff error due to hardcoded image tag as latest

    Generated skaffold config at /var/folders/r8/sq4khrmx1xnfc7rgqsznh1880000gn/T/skaffold_efl_nwvf/skaffold.yaml
    Generating tags...
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes -> ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:latest
    Checking cache...
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes: Found Locally
    Tags used in deployment:
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes -> ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:1fffdd3281571b184d412cd6cae6e7d727c24c20aa94c3c5c4666bde8d3e29e6
    Starting deploy...
    Loading images into kind cluster nodes...
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:1fffdd3281571b184d412cd6cae6e7d727c24c20aa94c3c5c4666bde8d3e29e6 -> Found
    Images loaded in 194.644416ms
    Helm release airflow not installed. Installing...
    Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
            * timed out waiting for the condition
    
    deploying "airflow": install: exit status 1
    

    [Q] build tag vs. deployment tag GoogleContainerTools/skaffold#6804

  2. Namespace not found error for Helm

    Stable repo is already added
    Good version of skaffold installed: 2.17.0 in /Users/jason/Desktop/airflow/.venv/bin
    Generating tags...
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes -> ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:latest
    Checking cache...
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes: Found Locally
    Tags used in deployment:
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes -> ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:1fffdd3281571b184d412cd6cae6e7d727c24c20aa94c3c5c4666bde8d3e29e6
    Starting deploy...
    Loading images into kind cluster nodes...
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:1fffdd3281571b184d412cdLoaded
    Images loaded in 1 minute 7.202 seconds
    Helm release airflow not installed. Installing...
    Error: INSTALLATION FAILED: create: failed to create: namespaces "airflow" not found
    Cleaning up...
    Error: uninstall: Release not loaded: airflow: release: not found
    Cleaning up resources encountered an error, will continue to clean up other resources.
    deploying "airflow": install: exit status 1
    

    skaffold not creating namespace for helm release GoogleContainerTools/skaffold#8515

  3. It's not an error, but the current --dags-path is default as dags related path instead of files/dags so we need to explicitly set to files/dags if we need to test out the same dags as breeze start-airflow.

    Stable repo is already added
    Good version of skaffold installed: 2.17.0 in /Users/jason/Desktop/airflow/.venv/bin
    Running skaffold without deploying Helm resources. If sync cannot find pods, rerun with --deploy.
    Generating tags...
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes -> ghcr.io/apache/airflow/main/prod/python3.10-kubernetes:latest
    Checking cache...
     - ghcr.io/apache/airflow/main/prod/python3.10-kubernetes: Error checking cache.
    getting hash for artifact "ghcr.io/apache/airflow/main/prod/python3.10-kubernetes": getting dependencies for "ghcr.io/apache/airflow/main/prod/python3.10-kubernetes": pattern "dags/**" did not match any file
    

What

To fix the above errors:

  1. Add setValueTemplates skaffold config to resolve image instead of hardcoding as latest image tag to avoid ImagePullBackOff error.
  2. Add missing deploy.helm.releases.createNamespace skaffold config
  3. change default value of --dags-path from dags to files/dags (align as breeze container mount point)

To make breeze k8s dev as one step command:

  • ask user whether to run create-cluster command if found that cluster hasn't been created.
  • if user run breeze k8s create-cluster command first, also show the hint of breeze k8s dev so that user don't need to run breeze k8s configure-cluster, breeze k8s build-k8s-image ... breeze k8s deploy-airflow.

…ion hints, and refine Skaffold config generation
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic !

@jason810496 jason810496 changed the title Refactor 'breeze k8s dev' command Refactor breeze k8s dev to enable one-step Kubernetes setup and fix Skaffold/Helm issues Dec 28, 2025
@jason810496 jason810496 changed the title Refactor breeze k8s dev to enable one-step Kubernetes setup and fix Skaffold/Helm issues Refactor breeze k8s dev to enable one-step Kubernetes setup and fix Skaffold issues Dec 28, 2025
@potiuk potiuk merged commit 33671b8 into apache:main Dec 29, 2025
128 checks passed
@github-actions
Copy link

Backport failed to create: v3-1-test. View the failure log Run details

Status Branch Result
v3-1-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 33671b8 v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

@potiuk
Copy link
Member

potiuk commented Dec 29, 2025

No need to backport

Subham-KRLX pushed a commit to Subham-KRLX/airflow that referenced this pull request Jan 2, 2026
stegololz pushed a commit to stegololz/airflow that referenced this pull request Jan 9, 2026
jhgoebbert pushed a commit to jhgoebbert/airflow_Owen-CH-Leung that referenced this pull request Feb 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants